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Abstract 


To  improve  operational  effectiveness  for  the  Canadian  Forces  (CF),  the  Joint  Unmanned  Aerial 
Vehicle  Surveillance  Target  Acquisition  System  (JUSTAS)  project  is  acquiring  a  medium 
altitude,  long-endurance  (MALE)  uninhabited  aerial  vehicle  (UAV).  In  support  of  the  JUSTAS 
project,  Defence  Research  and  Development  Canada  (DRDC)  -  Toronto  is  investigating  the 
human  factors  issues  of  UAV  ground  control  stations  (GCS)  interfaces  for  UAVs  and  exploring 
possible  solutions  using  multimodal  displays.  This  is  the  final  progress  report  summarizes  the 
project  from  September  of  2012  to  March  of  2013  of  this  Baseline  and  Multimodal  UAV  GCS 
design  and  participant  running. 
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Executive  summary 


Baseline  and  Multimodal  UAV  GCS  Interface  Design:  Progress 
Report 

Background:  Uninhabited  aerial  vehicles  (UAVs)  are  remotely  controlled  aircraft  used  for  a 
variety  of  civilian  and  military  applications  including  command,  control,  communications, 
computers,  intelligence,  surveillance  and  reconnaissance  (C4ISR).  To  improve  C4ISR  capability, 
the  Canadian  Forces  (CF)  is  acquiring  a  medium-altitude,  long-endurance  (MALE)  UAV  under 
the  Joint  Unmanned  Aerial  Vehicle  Surveillance  Target  Acquisition  System  (JUSTAS)  project.  In 
support  of  the  JUSTAS  project,  Defence  Research  and  Development  Canada  (DRDC)  -  Toronto 
is  investigating  the  human  factors  issues  of  UAV  ground  control  stations  (GCS)  interfaces  for 
UAVs  and  exploring  possible  solutions  using  multimodal  displays.  This  project,  the  Baseline  and 
Multimodal  UAV  GCS  Interface  Design,  comes  to  the  end  as  of  March  of  201 1. 

Results:  As  of  March  of  2013,  the  participant  running  for  both  the  baseline  and  multimodal 
condition  of  the  study  was  completed.  Several  issues  of  the  experimental  environment  were 
identified  and  resolved.  After  the  preliminary  data  organization  was  completed,  additional 
participants  were  run  to  achieve  the  designated  number  of  participants  in  the  design  of  the 
experiment. 

The  report  described  the  finalized  of  the  multimodal  displays  and  their  implementation  into  the 
multimodal  condition  of  the  experiment.  The  auditory  sonification  of  engine  RPM  and  auditory 
warnings  were  designed  and  introduced  to  the  experiment.  The  tactile  display  to  present  attitude 
upset  of  the  UAV  was  being  developed.  This  additional  experiment  was  completed  and  the 
promising  tactile  display  was  implemented  in  the  experiment. 

Significance:  The  report  discusses  the  completion  of  the  participant  running  for  both  the  baseline 
and  multimodal  conditions  of  the  experiment.  The  significance  of  the  multimodal  display  design 
and  the  implementation  of  the  multimodal  display  design  to  the  experiment  are  also  addressed. 

Future  plans:  The  project  comes  to  the  end  as  of  March  of  2013.  The  future  plans  of  the  project 
are  to  start  data  analysis  for  the  baseline  and  multimodal  condition  for  participants  to  explore  the 
significance  of  the  multimodal  display  in  the  UAV  GCS  scenarios. 
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1  Introduction 


An  Uninhabited  Aerial  Vehicle  (UAV)  is  an  aircraft  system  without  an  onboard  pilot  or  crew. 
The  UAV  is  controlled  from  a  Ground  Control  Station  (GCS).  Today's  UAVs  are  highly 
automated  and  to  some  extent,  autonomous.  UAVs  can  be  directed  to  follow  a  pre-programmed 
mission;  they  can  fly  to  designated  waypoints,  fly  specific  patterns,  correct  for  course  deviations 
and  hold  above  a  particular  coordinate  or  target.  Some  UAVs  can  perform  automated  take-off  and 
landing  (e.g.,  the  CU-170  Heron  used  by  the  Canadian  Forces).  UAV  developers  argue  that 
automation  and  autonomy  provide  several  benefits:  (a)  increased  flight  safety;  (b)  simplified 
operations;  (c)  lower  operating  costs;  and  (d)  reduced  operator  workload  (Attar,  2005).  However, 
these  benefits  are  not  always  realized.  Along  with  the  benefits  of  automation,  some 
disadvantages  occur  such  as  loss  of  situation  awareness  (Endsley  and  Kiris,  1995;  Endsley,  1996), 
loss  of  supervisory  control  (Parasuraman,  Molloy,  Mouloua,  &  Hilbum,  1996;  Sheridan,  1987), 
information  deprivation  that  occurs  from  remote  operations  (Manning,  Rash,  LeDuc,  Noback,  & 
McKeon,  2004),  and  high  workload  levels  for  operators  (Lee,  2008;  Woods,  1996).  These  issues 
point  to  the  need  for  improved  interfaces  to  help  these  operators  remain  in  the  loop  and  maintain 
situation  awareness  during  the  remote  monitoring  tasks  typical  of  UAV  monitoring. 

The  work  described  in  this  report  is  in  support  of  the  Canadian  Forces  (CF)  Joint  Unmanned 
Aerial  Vehicle  Surveillance  Target  Acquisition  System  (JUSTAS)  project.  The  JUSTAS  project 
entails  the  acquisition  of  a  medium-altitude,  long-endurance  (MALE)  UAV.  This  work  is 
directed  towards  understanding  the  human  monitoring  challenges  with  UAVs  similar  to  the 
UAVs  that  could  be  acquired  through  the  JUSTAS  project.  Further,  this  project  will  explore  the 
use  of  multimodal  interfaces  for  UAV  control  leading  to  new  design  criteria  for  UAV  ground 
control  stations,  or  improved  requirements  for  future  acquisitions. 

This  work  builds  on  the  work  of  Giang  et  al.,  2010  which  reviewed  the  current  research  on 
multimodal  displays  and  ecological  displays  for  UAV  control.  This  report  is  a  late-project 
progress  report  detailing  the  work  that  has  occurred  in  order  to  convey  participant  running  of  the 
two  GCS  designs.  One  of  the  GCS  designs  is  a  visual  interface  (called  the  “baseline  condition”), 
essentially  simulating  as  closely  as  possible  the  current  interface  for  the  Heron  UAV.  The  other 
design  is  a  multimodal  interface  where  tactile  and  auditory  information  will  be  added  to  the 
interface  to  see  if  this  new  information  can  improve  operator  performance  and  situation 
awareness. 

This  is  the  last  call-up  of  the  project.  This  work  occurred  between  September  of  2012  and  March 
of  2013  to  support  the  design  and  testing  of  a  multimodal  Uninhabited  Aerial  Vehicle  (UAV) 
Ground  Control  Station  (GCS)  simulator  with  novice  and  expert  participants.  This  report  will 
outline  the  progress  in  the  three  aspects  above.  In  September  of  2012,  the  evaluation  of  the  GCS 
Multimodal  Condition  was  started. 


2  Design  and  Implementation  of  the  Multimodal 
Displays 


As  of  March  of  2013  the  evaluation  of  the  multimodal  displays  for  the  GCS  Multimodal 
Condition  has  been  completed  but  the  data  has  not  yet  been  analyzed.  This  section  will  follow  the 
discussion  of  the  progress  report  Yeti  et  al.,  2012)  8148-07  and  describe  the  evaluation  of  the 
multimodal  displays  in  the  multimodal  condition  of  the  experiment. 

2.1  Implementation  of  the  Auditory  Display 

In  the  last  contract  call-up,  the  design  of  the  auditory  display  was  completed  to  integrate  the 
Tucker  Davis  Technologies  (TDT)  System  3  into  the  experiment.  An  auditory  circuit  was  built 
under  the  TDT  System  3  development  environment  to  produce  the  sonification  of  the  UAV 
engine  as  well  as  the  two  overlay  auditory  alarms.  Silent  gaps  have  been  added  between  two 
bursts  of  alarms  to  ensure  that  individual  bursts  could  be  distinguished.  During  the  period  of  this 
call-up,  the  auditory  circuit  has  been  adjusted  to  present  auditory  sonification  with  an  appropriate 
sound  level  to  the  participants  over  an  AKG  K501  headset.  The  signal-to-noise  ratio  of  the 
auditory  warnings  was  approximately  8  dB. 

2.2  Implementation  of  the  Tactile  Display 

A  3x3  grid  for  tactors  deployment  on  the  vest  was  implemented  as  the  tactile  display.  Prior  to 
September  2012,  an  experiment  was  conducted  to  examine  four  potential  tactile  display  designs. 
The  most  promising  design,  based  on  discriminability  and  appropriate  perceived  urgency 
mappings  was  implemented  as  the  tactile  display  in  the  multimodal  condition  of  this  experiment. 


3  Participant  Running  in  GCS  Multimodal  Condition 


From  September  of  2012  to  January  of  2013,  the  multimodal  interface  was  evaluated. 
Subsequently,  a  preliminary  data  cleansing  was  conducted  in  order  to  remove  extraneous  or  faulty 
data.  The  results  of  the  data  cleansing  showed  that  the  data  of  some  participants  was  corrupted, 
due  to  known  and  unknown  system  failures.  Therefore,  several  additional  participants  were 
recruited  and  run  from  January  of  2013  to  March  of  2013,  to  replace  those  participants  with  the 
corrupted  data.  In  this  section  the  procedures  and  events  of  the  multimodal  evaluation  have  been 
reported. 


3.1  Experimental  Protocol 

Each  participant  was  presented  with  an  information  package  and  was  asked  to  sign  a  consent  form 
prior  to  the  start  of  the  experiment.  In  August  of  2012,  the  consent  form  and  the  information 
package  were  updated  to  have  three  major  changes.  The  first  change  on  the  consent  form  and  the 
information  package  was  to  add  a  question  to  acquire  age  of  participants  during  the  experiment. 
The  intent  of  the  question  was  to  provide  more  precise  information  about  the  subject  group  for 
future  data  analysis  and  reporting.  The  question  on  age  was  added  to  the  existing  questionnaire  in 
the  training  session.  In  the  training  session,  participants  were  asked  to  complete  the  questionnaire 
after  two  practice  scenarios  were  played.  The  questionnaire  was  used  in  running  of  the  baseline 
condition.  The  second  change  was  to  include  facts  about  overall  sound  level  and  pressure  level 
during  the  experiment,  because  in  the  multimodal  condition  participants  were  tested  with  auditory 
and  tactile  signals.  The  third  change  was  to  revise  the  amount  of  remuneration.  To  allow  for 
testing  that  the  auditory  and  tactile  equipment  was  operating  properly,  the  length  of  the  first 
session  was  increased  by  one  hour.  Table  1  shows  the  comparison  of  remuneration  between  the 
baseline  condition  and  the  multimodal  condition. 


Table  1.  Comparison  of  remuneration  between  the  baseline  condition  and  the  multimodal 

condition 


Session 

Hourly  Rate 

Total  for  2-hour  Session 

Session  1 

$  10/hour 

$20 

Session  2 

$20/hour 

$40 

Session  3 

$30. 60/hour 

$61.20 

a.  Baseline  condition  (total  amount:  $121.20) 


Session 

Hourly  Rate 

Total  for  2-hour  Session 

Session  1 

$  13.40/hour 

$40.20 

Session  2 

$20/hour 

$40 

Session  3 

$30. 60/hour 

$61.20 

b.  Multimodal  condition  (total  amount:  $141.40) 

3.2  Experiment  Room  Setup 


Due  to  laboratory  renovations,  the  experimental  setup  was  located  in  a  different  room  from 
January  2012  until  May  2012.  During  this  time  period,  the  baseline  condition  was  run.  In  May  of 


2012,  the  original  experimental  room  was  fully  renovated.  In  September  of  2012,  the  participant 
running  of  the  GCS  Multimodal  Condition  was  started  in  the  original  experimental  room.  After 
the  completion  of  this  running,  all  the  additional  participants  were  also  taken  in  the  original 
experimental  room  (Figure  1).  Conditions  between  the  two  rooms  were  matched  as  closely  as 
possible. 


The  original  experimental  room 

The  alternative  experimental  room 

The  original  experimental  room 

Start  of  the  project;  all  tests,  pilot 
studies,  etc. 

GCS  Baseline  Condition 

GCS  Multimodal  Condition;  running 
of  additional  participants 

Figure  1.  The  GCS  experiment  rooms 

Similar  to  the  set-up  in  the  alternative  experimental  room  (where  we  completed  the  participant 
running  of  the  GCS  Baseline  Condition),  a  partition  was  set  in  the  original  experiment  room  to 
guarantee  the  minimal  disruption  of  the  experimenter  and  participant.  A  video  camera  was  placed 
in  the  Participant  Zone.  The  video  camera  only  monitored  the  computer  screens  of  the  GCS 
system  in  the  Participant  Zone.  The  experimenter  could  access  the  video  camera  in  the 
Experimenter  Zone. 


The  alternative  experimental  room 

Experimenter 

Zone 

z 

Participant 

Zone 

O 

3 

O 

Video 

Camera 

7 

Figure  2.  The  partition  in  the  GCS  experiment  room 


3.3  Modified  Participant  Recruitment  Process 


The  recruitment  of  participants  for  the  GCS  Multimodal  Condition  was  started  in  August  of  2012, 
prior  to  the  experiment.  The  experiment  was  conducted  in  two  participant  groups.  The  naive 
subject  group  consisted  of  participants  with  no  pilot  experience;  the  expert  subject  group 
consisted  of  participants  with  at  least  10  hours  of  pilot  experience.  We  used  the  same  methods  in 
recruiting  the  naive  participants  and  the  expert  participants.  Flowever,  the  participants  were 
required  to  self-declare  if  they  were  naive  or  expert  within  the  registration.  Such  recruitment 
methods  included  online  registration  webpages  and  posters  on  the  campus  of  University  of 
Waterloo  and  to  the  local  flying  club.  Wings  over  Waterloo.  The  recruitment  information 
received  ethics  clearance  by  Ethics  Office  of  University  of  Waterloo  prior  to  the  experiment. 

One  challenge  was  to  fit  the  participants  into  the  experiment  calendar.  Similar  to  the  GCS 
Baseline  Condition,  each  participant  was  required  to  come  in  for  three  sessions.  Flowever  in  the 
GCS  Multimodal  Condition,  the  length  of  each  session  was  not  equal  -  3  hours  to  complete  the 


first  session  and  2  hours  to  complete  either  of  the  second  and  third  session.  In  order  to  resolve  this 
problem,  an  online  scheduling  system  was  set  up  to  recruit  participants. 

To  register  for  the  experiment,  participants  were  asked  to  visit  the  online  scheduling  system.  First, 
participants  were  asked  to  complete  a  questionnaire  similar  to  Figure  3  to  ensure  they  were 
qualified  to  participate  in  the  experiment.  Participants  should  have  self-reported  normal  hearing 
and  have  no  discomfort  of  vibrotactile  signals  on  the  skin  of  their  back,  due  to  the  existence  of 
auditory  and  tactile  interfaces  in  the  GCS  Multimodal  Condition. 


Annex  A  Qualifications: 


1.  Are  you  an  expert  participant  (at  least  10  hours  of  pilot  experience)  or  a  naive  participant? 

•  Expert 

•  Naive 

2.  Normal  vision  is  required  to  attend  this  experiment.  Both  naked  eye  normal  vision  and  normal 
vision  with  contact  lens  are  acceptable.  Apologize  but  those  wearing  glasses  need  not  apply. 
Please  choose  one  of  the  following: 

•  Normal  vision  (naked  eye) 

•  Normal  vision  (contact  lens) 

•  None  of  above 

3.  Do  you  have  normal  heaming  ability?  You  will  be  asked  to  wear  a  headset  in  the  experiment. 

•  Yes 

•  No 

4.  Do  you  have  any  pain  or  discomfort  on  the  skin  in  your  back?  You  will  be  asked  to  wear  a  vest 
with  vibrotactile  units. 

•  Yes 

•  No 


Figure  3.  Questionnaire  for  qualifications 

After  the  qualifications  were  verified  by  the  experimenter,  participants  were  directed  to  a  calendar 
view  (Figure  4).  The  calendar  was  updated  in  real  time  to  show  the  current  available  time  slots. 
At  the  beginning  of  each  week  (Sunday),  the  experimenters  published  available  time  slots  for  the 
week.  Initially,  the  duration  of  each  available  time  slot  was  marked  as  3  hours  for  the  first  session. 
Up  to  three  time  slots  could  be  arranged  per  day,  which  were:  10:00am  to  1:00pm,  1:30pm  to 
4:30pm  and  5:00pm  to  8:00pm.  A  minimum  half-hour  gap  was  set  between  two  time  slots  to 
allow  equipment  preparation  and  data  backup.  The  three  time  slots  were  adjustable  in  special 
situations,  however  participants  were  asked  to  contact  the  experimenters  to  complete  such 
changes. 

Each  participant  was  asked  to  choose  three  empty  time  slots  from  the  calendar  to  book  his  or  her 
experiment.  The  first  session  which  took  3  hours  to  complete  was  booked  as-is.  However, 
participants  were  informed  that  the  second  and  third  sessions  were  subject  to  the  start  time, 
because  each  of  them  took  only  2  hours  to  complete.  When  the  booking  was  completed,  the 
experimenter  manually  modified  the  duration  of  the  second  and  third  sessions  from  3  hours  to  2 
hours.  All  sessions  were  expected  to  be  completed  within  one  week,  and  no  more  than  one 
session  could  be  booked  per  day.  The  online  booking  system  was  programed  to  ensure  the  two 
requirements  above. 


Figure  4.  The  participant  scheduling  method 


Participants  were  asked  to  complete  all  three  sessions  within  one  week.  This  made  scheduling 
difficult  with  three  time-slots  per  day,  and  only  15  possible  slots  each  week.  The  experimental 
procedure  was  changed  to  allow  for  three  sessions  within  7  days  instead  of  one  working  week  to 
make  scheduling  easier,  but  no-shows  and  cancellations  late  in  the  week  still  resulted  in 
scheduling  conflicts.  The  default  time-slots  on  the  calendar  were:  10am- lpm,  1:30pm-  4:30pm, 
and  5pm-  8pm.  The  time-slots  could  be  adjusted  regarding  the  special  need  of  participants,  but  all 
sessions  should  be  arranged  no  earlier  than  9am  and  no  later  than  9pm  due  to  security  reasons. 


Figure  5.  An  example  of  bookings 

3.4  Summary  of  Experiment  Completion 

From  September  of  2012  to  March  of  2013,  30  naive  participants  and  9  expert  participants 
successfully  completed  the  GCS  Multimodal  Condition  experiment  (Table  1). 


Table  2.  Participant  Status  of  the  GCS  Multimodal  Condition 


Number  of  Naive 


Number  of  Expert 


Participants 

Participants 

Expected 

30 

10 

Successfully  Completed 

30 

9 

Recruited 

34 

10 

Failed 

4 

1 

Failed  in  the  first  session 

3 

1 

Failed  in  the  second  session 

1 

n/a 

Failed  in  the  third  session 

n/a 

n/a 

Failure  Rate 

12% 

10% 

At  the  beginning  of  each  session,  the  participant  was  required  to  complete  an  eye  calibration  test 
to  ensure  the  participant’s  eye  movement  during  the  simulation  could  be  recorded  by  the 
eyetracking  system.  The  failure  of  any  eye  calibration  resulted  in  the  participant  being  requested 
to  terminate  the  experiment.  In  Table  2  above  this  is  reflected  as  a  “failure”.  The  total  failure  rate 
of  both  naive  participants  and  expert  participants  was  11%,  and  there  was  no  noticeable 
difference  of  the  failure  rate  between  naive  participants  and  expert  participants. 

3.5  System  Failures  and  Defects 


Overall,  the  GCS  system  was  stable  during  the  running  of  the  multimodal  condition.  All  the 
system  failures  that  occurred  during  the  experimental  sessions  in  this  call-up  were  known 
previously.  The  experimenters  observed,  diagnosed  and  handled  the  system  failures  by  consulting 
documented  procedures  established  when  running  the  baseline  condition.  The  types  of  failures 
are  documented  below. 

3.5.1  Scenario  Manager  Froze  on  Experimenter  Computer 


The  most  frequent  system  failure  was  a  crash  on  the  experimental  computer.  Participants  visually 
observed  the  following  popup  message  on  the  monitor:  "X-Plane  Software  has  encountered  a 
problem  and  needs  to  close".  All  log  files  up  to  that  point  were  kept  and  there  was  no  data  loss  up 
to  this  point. 

Reason:  Currently  unknown. 

Solution:  Confirm  data  had  been  saved  correctly.  If  no  events  occurred,  the  scenario  could  be 
restarted,  otherwise  we  used  only  data  collected  up  to  that  point. 


3.5.2  Replacement  of  the  Graphic  Acceleration  Card  on  Xplane 
Computer 


In  February  of  2013,  the  xplane  computer  experienced  a  fatal  failure  while  running  an 
experiment.  The  xplane  computer  crashed  a  few  minutes  after  each  time  it  booted.  The 
experiment  has  to  be  terminated  before  the  problem  was  solved. 

Reason:  On  further  diagnosis,  the  problem  was  determined  to  be  the  failure  of  the  cooling  fan  on 
the  graphic  acceleration  card  on  the  computer. 


Solution:  The  graphic  card  was  replaced.  As  the  exact  model  of  the  graphic  acceleration  card  was 
no  longer  available  in  the  market,  an  equivalent  model  provided  by  the  same  chipset  manufacturer 
was  used.  Before  the  experiment  was  resumed,  an  evaluation  was  taken  to  ensure  the  participants 
perceived  the  same  interface  visually  throughout  the  simulation  and  time  to  render  images  was 
equivalent. 

3.6  Preliminary  Data  Cleansing 

The  analysis  of  experimental  data  for  the  baseline  condition  and  the  multimodal  condition  of  this 
experiment  started  in  January  of  2013.  Each  participant  was  asked  to  complete  two  practice 
scenarios  and  twelve  formal  scenarios.  Only  the  twelve  formal  scenarios  of  each  participant  were 
analyzed.  Prior  to  the  analysis,  the  experimental  data  was  organised.  There  were  two  major 
reasons  to  proceed  with  the  data  cleansing.  First,  the  experimental  data  for  each  participant  was 
recorded  on  various  computers.  Secondly,  some  of  the  experimental  data  was  corrupted  due  to 
known  system  failures  which  have  been  discussed  in  the  last  contract  report  (call-up  8148-07). 
The  corrupted  data  were  replaced  by  data  collected  from  running  of  the  additional  participants, 
which  is  discussed  in  Section  4. 

At  the  time  of  writing  this  contract  report  (March  of  2013),  the  preliminary  data  cleansing  was 
ongoing.  The  first  step  in  the  data  cleansing  was  to  collect  data  from  the  three  computers  as 
presented  in  Figure  6  to  one  folder  named  by  the  identification  of  each  participant. 


•  Secondary  (Secondary  task  log  files) 

•  SA  (Situation  awareness  questionnaire  log  files) 

•  NASA-TLX  (NASA-TLX  questionnaire  log  files) 

Figure  6.  Data  organization 

There  were  several  methods  that  have  been  used  to  clean  the  data: 

1 .  General  log  files  for  each  scenario 

The  general  occurrences  of  each  scenario  were  described  by  several  log  files  which  were 
recorded  on  the  experimental  computer.  The  experimental  computer  was  controlled  by  the 


experimenter  during  the  sessions  to  load,  to  start  and  stop  the  scenarios.  The  Event  log  file 
recorded  all  the  important  events  occurred  during  the  presentation  of  each  scenario,  including 
the  occurrence  of  each  critical  event.  The  event  log  file  also  collected  participant’s  response 
to  such  event,  such  as  pressing  of  the  “operator  concern”  button  and  the  “abort”  button. 
Another  important  variable  coming  from  the  event  log  file  was  the  presentation  of  tactile 
stimuli.  In  the  file,  the  magnitude  of  the  vibration  was  recorded.  By  March  of  2013,  data 
cleansing  for  the  event  log  file  has  been  started,  while  the  other  two  types  of  log  files  (UAV 
log  file  and  eye  tracking  file)  have  not  been  started  yet. 

2.  Eye  tracking  raw  data  and  video  clips 

The  eye  tracking  raw  data  was  recorded  and  stored  on  the  xplane  computer,  where  the 
participants  operated  the  GCS  simulator.  Data  cleansing  for  such  data  has  not  been  started  by 
the  time  of  writing  this  report.  Video  recording  of  the  desktop  of  the  xplane  computer  was 
also  located  on  the  xplane  computer.  The  video  recording  showed  exact  the  same  display  the 
participants  were  presented  throughout  the  experiment.  The  cleansing  of  this  type  of  data  has 
been  started  on  January  of  2013. 

3.  Secondary  task  data  and  questionnaire  data 

The  cleansing  of  the  secondary  task  data  has  started.  In  each  scenario,  the  secondary  task  data 
began  shortly  after  the  UAV  was  launched,  and  the  participants  were  asked  to  response  to  the 
secondary  task  monitor  during  the  scenario.  However,  it  has  been  found  from  the  video  clips 
that  during  the  occurrence  of  the  “xplane  encountered  an  error”,  which  was  a  known  system 
failure,  an  error  dialog  partially  obscured  the  map  area  of  the  screen.  This  error  may  have 
disturbed  participants  while  performing  the  secondary  task.  Thus,  all  the  secondary  data 
(a.k.a.  responses  from  participants)  was  removed  during  the  presentation  of  the  error  dialog. 


4  Additional  Participants 


After  the  completion  of  the  participant  running  of  the  GCS  Multimodal  Condition,  the 
experimental  data  was  cleaned  and  organized.  More  details  of  this  process  can  be  found  in 
Section  3.  The  results  of  the  data  cleansing  showed  that  data  files  of  some  participants  were 
corrupted  due  to  system  failures.  Several  additional  participants  were  recruited  and  run  to  replace 
participants  whose  data  was  seriously  corrupted. 

Table  3.  Additional  Participant  Running  Status 


GCS  Baseline  Condition 

GCS  Multimodal  Condition 

Nave 

Expert 

Naive 

Expert 

Expected 

4 

2 

5 

- 

Successfully  Completed 

4 

2 

5 

- 

Recruited 

5 

2 

6 

- 

Failed 

1 

- 

1 

- 

Failed  in  the  first  session 

1 

- 

1 

- 

Failed  in  the  second  session 

- 

- 

- 

- 

Failed  in  the  third  session 

- 

- 

- 

- 

Failure  Rate 

20% 

- 

17% 

- 

Table  2  shows  the  summary  of  running  additional  participants  for  the  baseline  condition  and  the 
multimodal  condition.  No  expert  participants  were  involved  in  the  multimodal  condition,  because 
the  restriction  of  data  cleansing  progress  as  well  as  the  lack  of  sources  for  recruiting  expert 
participants. 


5  Summary 


In  summary,  considerable  progress  has  been  made  between  September  of  2012  to  March  of  2013. 
The  running  of  participants  in  the  GCS  multimodal  condition  has  been  completed,  and  the 
preliminary  data  analysis  has  been  started.  Running  of  the  additional  participants  for  both  the 
baseline  condition  and  the  multimodal  condition  was  also  completed. 
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List  of  symbols/abbreviations/acronyms/initialisms 


CF 

Canadian  Forces 

DND 

Department  of  National  Defence 

DRDC 

Defence  Research  &  Development  Canada 

DRDKIM 

Director  Research  and  Development  Knowledge  and  Information 
Management 

GCS 

Ground  Control  Station 

HFES 

Human  Factors  Ergonomics  Society 

HREC  or  DRDC 
HREC 

Defence  Research  and  Development  Canada  Human  Research  Ethics 
Committee 

HUD 

Heads  Up  Display 

JUSTAS 

Joint  Unmanned  Aerial  Vehicle  Surveillance  Target  Acquisition  System 

MALE 

Medium-altitude,  Long-endurance 

MAUVe 

Multiple  Autonomous  Unmanned  Vehicle  Experimental 

NASA-TLX 

NASA  Task  Load  Index 

ORE  or  UW 

ORE 

University  of  Waterloo  Office  of  Research  Ethics 

RPM 

Revolutions  per  minute 

RPvdsEX 

Real-Time  Processor  Visual  Design  Studio 

R&D 

Research  &  Development 

SA 

Situation  Awareness 

SOW 

Statement  of  Work 

UAV 

Uninhabited  Aerial  Vehicle 

